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ABSTRACT 

In engineering education, assessment instruments are often developed to evaluate programs 
and projects. Unfortunately, these innovations are not always adopted by intended audiences. 
Rogers’ Diffusion of Innovations (Dl) Theory provides a framework to analyze characteristics of 
an innovation that will affect adoption. The Appraisal System for Superior Engineering Education 
Evaluation-instrument Sharing and Scholarship (ASSESS) is a user-driven, web-based catalogue 
of assessment instrument information. The purpose of this study is to develop an understanding 
of ASSESS’ adoptability as well as to characterize lessons learned and how to apply them to other 
technical innovations. Interviews with potential users were used to explore user perspectives of AS¬ 
SESS. It was found that the innovation’s Use as an Alternative and its Functionality of Design were 
the primary categories important for adoption. These categories relate to the Relative Advantage, 
Complexity, and Compatibility Dl characteristics. Focusing on these categories and Dl characteristics 
is recommended for developers of engineering education technological innovations. 
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INTRODUCTION 

Educational innovations to improve and assess engineering education are often developed, but 
not commonly adopted by the intended users. An abundance of assessment instruments, such as 
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concept inventories and self-efficacy instruments, have been developed for engineering education. 
However, these instruments are not broadly adopted. Rogers, who developed the Diffusion of In¬ 
novations Theory (Dl), notes that even innovations with obvious advantages are slow to be adopted 
in educational institutions (Rogers 2003). 

Adoptability studies are usually performed after the development process and typically study how 
an innovation is adopted (Rogers 2003, pg. 219). One issue with studying adoptability by looking at 
the adopters is that at this point it is too late to make significant improvements or changes to the 
innovation to aid its adoption. Studying the characteristics of an innovation can be done before the 
innovation is released for adoption, allowing for improvements to be made. This study will investigate 
potential adopters’ views of a web-based database developed to house information on engineering 
assessment instruments during the development process, allowing for insights into adoptability of 
technological innovations as they are developed and providing feedback to the developers for im¬ 
proved adoption. The paper has implications for engineering educators, researchers, and evaluators 
intending to adopt assessment instruments for improved classroom performance or for measuring 
progress toward goal attainment in evaluation of research innovations. 

LITERATURE REVIEW 

When new innovations are created, developers may be interested in how they will diffuse to their 
intended audience. This raises the question of how to determine the adoptability of the innovation. 
This information is important when defining the success of the innovation (Procaccino et al. 2005). 
One common method for understanding the adoptability of an innovative technology, especially in 
an educational setting, is Rogers’ Dl Theory (Martins et al. 2004, Kebritchi 2010). Kebritchi (2010) 
notes that “ Rogers’ theory has been widely used in a variety of settings, ranging from diffusion of 
rural technology among farmers to the adoption of innovations in educational settings. This theory 
is useful for instructional technologists to explain, predict and account for the factors that impede 
or facilitate the diffusion of their products” (p. 258). 

Rogers’ theory follows the diffusion of an innovation, looking at characteristics of an innovation, 
the social system in which it is diffused, the type of innovation decision, the types of communication 
channels used, and any promotional efforts used by change agents (Rogers 2003, pg. 11). Most of 
these variables occur after an innovation is released for adoption. The Dl characteristics of the innova¬ 
tion itself can be addressed during the development phase of the innovation. Because this research is 
focused on the development phase, this paper will use Dl characteristics to examine adoptability. 

Dl Theory suggests that five characteristics relate to the adoptability of an innovation, as 
summarized in Table 1 below (Rogers 2003, pg. 15-16). Dl Theory states that “ The characteristics of 
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Characteristic 

Definition 

Relative Advantage 

Compatibility 

Complexity 

Trialability 

Observability 

The degree to which an innovation is better than an existing method/practice/idea. 

The degree to which an innovation matches the needs, experiences and views of the adopter. 

The degree to which an innovation is perceived as difficult to use or understand. 

The degree to which the innovation may be experimented with before committing to adoption. 

The degree to which the results of an innovation are visible to others. 

Table 1. Diffusion of Innovations Characteristics 


innovations, as perceived by individuals, help to explain their different rates of adoption” (Rogers 
2003, pg. 15). The “individuals” mentioned are potential adopters, not the developers of the innova¬ 
tion. Most adoption studies neglect the perspective of the developer all together. Developers often 
have little or no awareness of what potential users are looking for in an innovation (Vonk et al. 2007). 
Ravitz and Hoad ley point out that “ One gap [in communication] is between technology users and 
technology developers, making it more difficult for developers to seek and obtain feedback from 
users” (Ravitz and Hoadley 2005). This study is unique in that developers will be immediately aware 
of potential users’ views of the adoptability of the innovation. 

There is disagreement in Dl literature about which of Rogers’ characteristics are the most important 
for the adoption of technological innovations (Kebritchi 2010, Martins et al. 2004, Samarwickrema and 
Stacey 2007). Literature about adoption studies for technological innovations in education helped 
determine what Dl characteristics would be important for this study. Studies that operationalized 
the five Dl characteristics for use as interview questions were specifically explored and the resulting 
three studies are described below. 

Kebritchi studied the adoption of a mathematical computer game (2010) targeted to middle school 
teachers to help their students learn Algebra. The Dl characteristics that were the most important 
were Relative Advantage, Compatibility, Complexity and Trialability. Relative Advantage was the most 
important factor, because the game had to be more effective at teaching Algebra than other options 
available to the teachers. Compatibility was also highly important because teachers needed the game 
to fit well into their schedules and curricula. If the game could not be adjusted to their requirements, it 
would not be helpful. Complexity was important for teachers in respect to how the game challenged 
their students. They did not want a game that was too easy, but if it was too challenging it would not be 
useful. It also had to be simple enough for the teachers to help their students if they ran into difficulty. 
Finally, teachers wanted opportunities to test the game before they committed to adopting it. Without 
at least minimal Trialability, few teachers were willing to implement the game. Observability was not 
important because teachers wanted to base adoption on their own experiences. 
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In researching the adoption of Internet resources for teaching foreign language in schools, Martins 
et al. found that Trialability and Observability were the most significant Dl characteristics (2004). Their 
study showed that participants who received more training on using the Internet for this purpose adopted 
the resource in their classroom. They found that when the applications were observable, diffusion was 
more likely to occur. The other three Dl characteristics were not perceived as important as Trialability 
and Observability. During training teachers mentioned the advantages of using the innovation (Relative 
Advantage ) and the ease of use ( Complexity ) were apparent. Thus, Relative Advantage and Complexity 
played a role in adoption, but were actually initially considered part of Trialability by the researchers. 

Another adoption study was focused on web-based learning and teaching in higher education 
(Samarwickrema and Stacey 2007). Relative Advantage was discovered to be the most influential Dl char¬ 
acteristic followed by Compatibility. The features of the web-based approaches, such as communication 
options, were often cited as reasons for adoption. Adopters previously did not use web-based learning, 
so the adoption of these methods was an alternative to classroom education. The study found that when 
the participants’ social systems encouraged them to adopt, they were more likely to do so. Because the 
systems were compatible with their values and those of their social network, they were comfortable with 
adoption. Trialability, Observability, and Complexity did not affect adoption as much. This could be be¬ 
cause most adopters were adopting based on the end goal (web-based learning) instead of based on the 
learning management system being recommended. If the instructor wanted to provide a web-based class 
(or was asked to do so by administration), then these three Dl characteristics became less important for 
adoption to occur. The advantages of the system and whether they are compatible with the educational 
environment are important. Whether or not the adopter’s colleagues had used the system, the system 
was simple, or a free trial was offered, were found to be less important for adoption. 

The significance of the various Dl characteristics on the adoptability of an innovation appears to 
be distinct for each innovation. Rogers himself stated that Relative Advantage is one of the strongest 
predictors of an innovation’s rate of adoption, but not all literature is in agreement (Rogers 2003, 
pg. 233). Thus, developers are not able to select a Dl characteristic for focus prior to initiating an 
adoptability study on their innovation. 


ASSESS 

Advancing engineering education requires development and use of high quality educational 
evaluation instruments as well as a community of engineering educators prepared to properly 
implement these instruments. Many of the research questions addressed by engineering education 
development projects require both qualitative and quantitative methods of assessment to determine 
project impacts on performance, perceptions, and academic intentions of people being studied. 


4 


WINTER 2014 





ADVANCES IN ENGINEERING EDUCATION 

Adoption of Technological Innovations: A Case Study of the ASSESS Website 



Project investigators often struggle to find an appropriate evaluation instrument, so they develop 
their own. This often results in use of less proven evaluation instruments or wasted effort as instru¬ 
ments are needlessly reinvented. Many investigators also lack the knowledge needed to select the 
best evaluation instruments, administer them as intended by the developer, and interpret evaluation 
results properly for the instrument and the conditions of its use. The final result is project evaluation 
that is inadequate for fully documenting the outcomes, guiding project implementation for maximum 
benefit, and determining real impacts of a given engineering education development project. 

At a time when improvement of engineering education is crucial to the well-being of the nation, 
high quality evaluation instruments must be readily available and also empower project investigators 
to select and utilize them effectively. The National Science Foundation is also calling for evaluation 
capacity building (ECB) for personnel working on NSF projects (Katzenmeyer and Lawrenz 2006). 
A national workshop convened by the authors of this paper also produced recommendations calling 
for a well-developed clearinghouse for high-quality evaluation instruments to support engineering 
education (Davis et al. 2009). Thus, a national imperative exists to develop, make readily available, 
and properly use high quality evaluation instruments in engineering education projects to support 
vital advancement of engineering education. 

Several online warehouses aim to address the need for more easily accessible assessment instru¬ 
ments. For example, Purdue’s INSPIRE Assessment Center offers engineering education researchers, 
outreach developers, and P-12 teachers psychometric and descriptive information on approximately 
60 research and classroom assessment instruments (see: Fttps ://engineering.purdue.edu/lnspire I 
Icenter/assessment-centeil) . The Assessing Women and Men in Engineering (AWE), developed by 
The Pennsylvania State University and the University of Missouri with supplemental funding from 
the Henry Luce Foundation, offers STEM assessment resources for K-16 educators, researchers, 
and evaluators (see: http://www.engr.psu.edu/awe/D - Engineering Pathway offers a variety of K-12 
and higher education teaching and learning resources (see: http://www.enqineerinqpathwav.orql) . 
Other websites, such as the Online Evaluation Resource Library (OERL), http://oerl.sri .com71, and 
the Field-tested Learning Assessment Guide (FLAG), http://flaquide.orgl, offer strategies and tools 
for classroom assessment and project evaluation. The websites all address different user types and 
vary in levels of comprehensiveness; yet, offer some form of education on incorporating assessment 
or evaluation into research or classroom-based practices. 

This study will examine the adoptability of the Appraisal System for Superior Engineering Education 
Evaluation-instrument Sharing and Scholarship (ASSESS), a user-driven, web-based database of evaluation 
instrument characteristics. While similar in nature to the aforementioned assessment resource websites, 
particularly INSPIRE, ASSESS houses more comprehensive psychometric and descriptive information for 
more than 100 instruments and is designed to meet the assessment and evaluation needs of engineering 
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Search Engine for Assessment Instruments 


'ASSESS 

Engineering Education 


Log in Register 

Why Register? 


ASSESS 


Engineering Education 


Search 

Search for instruments meeting your needs 

Learn 


Search Engine for Assessment instruments 


Leam about assessment evaluation as applied to engineenng 
education 


Find trustworthy help for: 

• Assessing engineering education innovations 

• Documenting engineering student learning attitudes, perceptions 

• Explaining your engineenng outcomes assessment 


Rate 

Provide your rating and comments on ASSESS features and 
resources 


ASSESS' helps you identify and judge suitability of instruments to meet the diverse assessment and 
evaluation needs in engineenng education. 


ASSESS is the Appraise/ System for Superior Engineering Education E/atoation-nstrwnent Sharing and Scholarship, 

■ developed mith funding from the National Science Foundation grant: DUE 1065486. Any opinions, findings, and 
conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the 
views of the National Science Foundation. 


Propose 

Propose additional instrument candidates for ASSESS 


Get Help 

Find FAQs and instructions for using ASSESS 


• ASSESS is a tool for the engineering education community to increase its ability to 
evaluate student learning and other impacts from educational practices and 
innovations. 

• The purpose of ASSESS is to help users identify and judge the suitability of 
different assessment instruments for their specific needs in engineering education. 


Figure 1. Introduction to ASSESS. Link: https://www.youtube.com/watch?v=RmU4H0izrYA&i 
ndex=6&list=PL84VcUp5xaBsRutUS-8KR2lwFxh8Kfl9G 


educators, researchers, and evaluators. For each assessment instrument, ASSESS provides descriptions, 
uses in engineering education, reliability and validity evidence, as well as both expert and user reviews 
that are unique to ASSESS. An introductory video describing ASSESS is available by clicking on Figure 1. 
The ASSESS website can be found at: http://assess.tidee.org4 Links to each of the previously mentioned 
assessment websites can be found by clicking on Resources under the Learn tab. 

In considering the purpose and functionality of ASSESS and the literature related to the most 
important characteristics for adoption of comparable innovations, it appears that Relative Advan¬ 
tage, Trialability, and Compatibility will be the most important characteristics related to adopt¬ 
ability of ASSESS. This innovation is different from those presented in the literature review in that 
it is specifically intended for engineering education, as opposed to education in general. ASSESS 
allows for user input, a function the reviewed innovations did not have. Additionally, ASSESS serves 
as a database of information with search functions and additional opportunities for learning. How¬ 
ever, all innovations included in the literature review are technological in nature and are used in 
education. One of the goals of this study is to compare the predicted and actual importance of Dl 
Theory Characteristics. 
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The goals of the ASSESS project are to assist engineering educators in identifying assessment 
instruments suitable for use in specific education or research endeavors. The project also seeks to 
assist engineering educators in disseminating and refining assessment instruments that have been 
developed through research projects. ASSESS users are able to submit instruments for consideration 
for inclusion through the ASSESS interface. 

ASSESS will potentially be used by teachers, program assessors and individuals writing grants and 
looking for information on assessment instruments. The development of ASSESS was chosen because 
it fills the need in engineering education for a source of information about assessment instruments 
and has a high probability of being adopted by large numbers of engineering educators. Finally, AS¬ 
SESS takes the form of a database, which could be translated to areas in education that store other 
materials, such as curricular materials or assessment questions. Thus, the results of this study may be 
helpful to an audience much broader than those interested in the specific content area of ASSESS. 


RESEARCH GOALS 

Literature shows much disagreement on what Dl characteristics are most important for the 
adoption of an innovation, a challenge for developers who would like to collect feedback on their 
innovation to improve adoptability. Adoptability studies need to be performed during the develop¬ 
ment process to allow developers to make adjustments to their innovations. 

The purpose of this study is to demonstrate the perspectives of potential ASSESS users, gathered 
using the Dl theory as a framework. The following research sub-goals will accomplish the overall 
research goal. 

• Capture multiple perspectives of the innovation during the development phase to develop 
and refine ASSESS. 

• Determine the relative importance of Dl characteristics in technological educational innovations. 


METHODOLOGY 


Participant Selection 

The participants were chosen based on their likelihood to adopt ASSESS, in a similar fashion to 
Guido Vonk in his study of planning support systems (Vonk, Geertman, & Schot, 2007). The first group, 
Participant Group (PG) One, is individuals who participated two years earlier in a workshop focusing on 
defining the need for and content of a database for engineering evaluation instruments. Everyone who 
participated in the workshop was invited to participate in this study. Of the 29 invited to participate, 
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nine agreed to be interviewed. PG One had already been exposed to the idea of ASSESS, but not the 
website itself. Since this group had chosen to attend the workshop and were engaged in engineering 
education and assessment, they were deemed likely potential adopters of ASSESS. PGs Two and Three 
were selected from the Educational Research and Methods (ERM) Division of ASEE. PG Two included 
nine individuals and PG Three included four individuals. ERM members consist of engineering educators 
and researchers who engage both in basic engineering education research and evaluation of their own 
and others’ teaching practices. Both researchers and educators are the targeted audience for ASSESS. 
Many ERM members attend educational conferences where ASSESS has been and will again be pre¬ 
sented. Their potential increased exposure to the ASSESS project from these conferences and the fact 
that they are interested in educational research made this group more likely to adopt the final innovation. 
Additional information about the ASEE ERM division can be found at: http://erm.asee.or S3 

The three groups of participants were interviewed throughout the development process of 
ASSESS to form a complete perspective on adoptability. The interviews occurred after the initial 
prototyped site was completed, but before the website was in its completed stage. For PG One, 
ASSESS had basic functionalities and about 10 instruments with information. For PG two, ASSESS 
had the same amount of instruments with information, more search features, and outlines of ad¬ 
ditional pages (learn, rate, etc.). For PG three, some additional instrument information had been 
added, the additional pages had some information, and improvements had been made in the search 
features’ functionality. Having the interviews staged at different times enabled the researchers to 
gather perspectives on changes made during the development process, allowing the development 
team to see what improved or what still needed to be worked on. 

Interview Protocol 

Interview questions were developed using the five Dl characteristics as guidelines. Each of the Dl 
characteristics had multiple questions associated with it, as shown in Table 3. In addition to these 
questions, general questions were used to determine the interviewee’s background with assess¬ 
ment instruments. The interview questions were also developed, in part, based on previous studies 
that had operationalized the Dl characteristics (Martins et al. 2004, White 2010, Kebritchi 2010, 
Samarawickrema 2007, Hill and Lee 2010, Liberatore and Breem 1997, Di Benedetto et al. 2003, Heri 
and Mosler 2008). However, Rogers discourages reusing other studies’ questions as “The specific 
ways in which the five attributes are expressed differs in each study, and so the measures of these 
attributes should be uniquely created afresh in each investigation” (Rogers 2003, pg. 225). For each 
interview, the participants were also given the definitions of the five Dl characteristics and were 
asked to describe their perspective of ASSESS demonstrating each of the Dl characteristics. The 
wording of some of the interview questions was slightly modified after completing the initial group 
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of interviews to increase clarity and optimize data collection from participants’ interviews. Table 2 
provides a summary of the questions asked for the participant interviews and the Dl characteristic 
they illustrate. Generally speaking, fewer questions were asked about Trialability and Observability. 
During the protocol development and the interviews participants had less to say about these charac¬ 
teristics, even when asked multiple probing questions. Therefore, the reduced number of questions 
does not represent a reduced opportunity to talk about these characteristics by the participants. 


Interview Methodology 

The goal of the interviews was to develop an understanding of the participants’ views of ASSESS’ 
adoptability using the Dl framework. Interviews were semi-structured (Krathwohl 1998, pg. 287) with 
open-ended questions (Patton 2002, pg. 20-21). A semi-structured interview style was selected based 
on the complexity of responses from individuals. The ability to individualize the interviews to collect the 
participants’ unique perspectives is critical when it is expected that aspects of each participant’s per¬ 
spective will be unique. The researchers did not want to assume that they could get all the information 
they needed from using a fixed set of questions, as would be the case in a closed-ended interview. 

Participants were asked the questions presented in Table 2 and were encouraged to elaborate 
on their answers. Follow up questions (shown italicized in Table 2) were asked as necessary 



Background Questions 

• What current methods do you use to access assessment instruments? 

• What do you like/dislike about your current methods? 

• What do you think of the ASSESS database? What was your first impression of the database? 

Does it meet your expectations or not? How so? 

Relative Advantage 

• What advantages or disadvantages does this database have over your existing methods for finding 
assessment tools? 

• In what ways would this database positively or negatively affect your assessment capabilities? 

• How would your assessment practices change if you were using this tool? Would you consider 
this change an improvement or a regression? 

• How would your productivity change due to the use of this tool? 

Compatibility 

• How do you envision ASSESS fitting into your daily work activities? 

• Why would you want to tell your colleagues about this? What would you tell them? 

• In what ways could this be used in your department? 

• In what ways could you envision your colleagues in your department using this tool? 

Complexity 

• When looking at the ASSESS database, what items, if any, do you not understand? 

• How hard would it be for you to learn to use this tool? 

• How long would it take you to become comfortable using ASSESS? 

• How easily would you be able to get what you want out of it in an efficient manner? 

Trialability 

Observability 

• What do you think the upsides/downsides to trying this tool are? 

• What are you giving up/gaining from trying ASSESS? 

• What other databases, similar in educational content to ASSESS, do you know of? What do you 
know about them and where did you hear about them? 

• How would others know if you were using this tool? 

Table 2. Participant Interview Questions 
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for clarification. The interviews were conducted over the telephone and lasted between 10 and 
20 minutes. 


ANALYSIS METHODS 

The interviews were recorded, transcribed, and analyzed using Atlas Tl (Scientific Software 
Development 1999). Although the three sets of interviews were conducted at different times during 
the development of ASSESS, there were not dramatic differences in participants’ responses across 
the three groups. Therefore, the data from all three groups were analyzed together. Data analysis 
was done with increasing depth as described by Krathwohl (1998, pg. 308). The transcriptions were 
first read to obtain an understanding of the overall responses. An initial coding step was then com¬ 
pleted, which is referred to as descriptive coding to ‘‘see what is there ” (Krathwohl 1998, pg. 308). 
The data was coded for participant responses that mapped to one of the five Dl characteristics. 
The next step of coding was to “recode this same material at an interpretive level” (Krathwohl 1998, 
pg. 308). The responses were then coded as they related to the Dl characteristic - good or bad. 
The final step for analysis consisted of “a still deeper level of analysis....code patterns of activities, 
themes, causal link...” (Krathwohl 1998, pg. 308). 

The responses were then categorized further with individual tags for the reasoning behind the 
categorization. For example, a participant’s response that said they liked the database because 
they found it self-explanatory and easy to use would be coded as “Complexity - Good - Simple” 
because it demonstrated good (low) levels of Complexity because of how simple it is to use the 
interface. The dataset was then recoded according to descriptions of the site that are more in terms 
of the interviewees’ language, both for interpretive value and to facilitate transferring findings to the 
developers. For example, responses were coded as “Additional Features” or “Suggested Changes” 
to help with the development process. The data were analyzed iteratively to connect the intervie¬ 
wees’ descriptions with the Dl characteristics. For example, the description of available information 
is related to Dl characteristics Relative Advantage, Compatibility, and Complexity. Finally, for each 
category of overlap between interviewees’ descriptions and Dl characteristics the approximate fre¬ 
quency of this occurrence was determined and characterized as high, medium, or low and as either 
positive or negative. If less than 25% of participants mentioned the category a “low” ranking was 
assigned, between 25% and 75% received a “medium” ranking, and 75% received a “high” ranking. 
For example, “Negative High” indicates that most responses were negative and greater than about 
75% of participants believed this category was important. Greater precision in these levels is not 
possible due to the inherent challenges and potential misrepresentation in assigning numerical data 
to interview responses, coupled with the limited value in doing so. 
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Data from PG Three did not reveal substantial new information beyond the previous interviews, 
known as data saturation. During analysis, 37 codes were used for PG Two, 36 of these were also 
used in PG Three. No new codes were added in analyzing PG 3. Krathwohl defines data saturation as 
the point “when new observations cease to add much to previous ones” (Krathwohl 1998, pg. 260). 
Therefore, no additional interviews were conducted after PG Three, because no new observations 
were being added. 


RESULTS AND DISCUSSION 

The results are presented in terms of six categories that were identified in terms of interviewees’ 
language: Expectations of Instrument Inclusion, Available Information, Community Features, Use 
as an Alternative, Searchabi/ity and Functionality of Design. This allows for a more reader-friendly 
presentation because it is an organization and representation in the terms and the words of po¬ 
tential users, and it is expected that readers will more likely understand categories representing 
these potential users than the Dl characteristics that most will not be familiar with. Also, this 
choice provides a more reader-friendly presentation of videos embedded into the article based 
on the six categories. These six categories, and the Dl characteristics they relate to, are shown in 
Table 3 below, with a “Positive” or “Negative” sign indicating if overall responses were positive 
or negative. If no responses related to a Dl characteristic for one of the six categories, this space 
was left empty in Table 3. The discussion will focus first on each category and then on how each 
Dl characteristic related to that category. Videos are also included to provide an overview of the 
functionality of ASSESS and features that resulted from or were already aligned with feedback 
from interviewees. 



Expectations of Available 

Instrument Inclusion Information 

Community 

Features 

Use as an 
Alternative 

Searchability 

Functionality 
of Design 

Relative Advantage Negative High 

Positive Med 

Positive Med 

Positive High 


Positive Med 

Compatibility 

Negative Low 

Positive Med 

Positive Low 

Positive Med 

Positive Med 

Negative Low 

Complexity 


Negative High 



Positive Low 

Positive Med 

Trialability 




Positive High 

Negative Low 

Positive Low 

Observability 



Positive Low 

Positive High 


Negative Med 


Table 3. Ratings for Dl Characteristics 
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Expectations of Instrument Inclusion 

The first category is “Expectations of Instrument Inclusion” and is related to expectations of inter¬ 
viewees to find information on specific instruments. This category includes aspects of the Relative 
Advantage and Compatibility Dl characteristics, as shown in Table 3. 

Relative Advantage - Participants expressed concern about the database being comprehensive: 
“Downsides, like with all repositories, it’s only as good as the effort to be inclusive of getting things 
into it,” “there’s also kind of a concern about would it be updated over time,” and “The other thing is 
/ tend to not only look at instruments that have only been done in engineering because / like looking 
at a wide variety of fields.” This concern is especially understandable considering the low number 
of instruments that had information available in the database at the time of the interviews. 

While the Relative Advantage Dl characteristic did receive a high negative rating for Expecta¬ 
tions of Instrument Inclusion, not all of the responses were negative. Some participants looked 
beyond the current size of the database and considered the end result: “I think the advantage is 
that it [ASSESS] is engineering focused and we are an engineering school so it filters out...things 
that / don’t want.” 

Compatibility - Participants were concerned that the database might not include information on 
the types of instruments they would be interested in: “I think there would need to be more focus 
on concept inventories and things of that nature, you know, specific skill-based as opposed to at¬ 
titude and things of that nature,” and “I guess for me the really emphatic point here is I’d like to see 
more qualitative instruments such as open ended surveys and guided survey questions and how to 
conduct a focus group.” 

In an effort to alleviate the concerns found in this category, the development team requested 
instrument suggestions from interview participants and has begun requesting them from conference 
attendees where the ASSESS website is being presented. The ASSESS website, once fully developed, 
will also have a feature that will allow users to suggest or submit instrument information to be in¬ 
cluded in the database. Thus, the database should evolve to be a highly inclusive set of instrument 
information providing for each user’s individual needs. 

Available Information 

Available Information includes the participant perspectives of the information available on the 
ASSESS website. Examples of information are: specifications about instruments, information about 
how to use ASSESS, and information about assessment instruments in general. These responses 
relate to the Relative advantage, Compatibility, and Complexity Dl characteristics. Table 4 shows 
that Relative Advantage and Compatibility received medium positive ratings and that Complexity 
received a high negative rating. 
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Relative Advantage - Participants overall were pleased with the amount of information avail¬ 
able: “ Despite the fact that i need it or don’t need it you have more details on the instrument. 
You have summary description, you have the reference...where it was published, reliability and 
validity, you will have those links.” Additionally, participants thought that they would be able to 
learn from the information on ASSESS. These two areas increased the overall Relative Advantage 
for potential users. 

Compatibility - Participants’ views of the available information were generally positive. Again, 
the amount of information available was the most significant topic of responses: “ And if i dive into 
one in particular it’s nice, you get the description, you get all the specifications of the material. You 
can link things, I can probably download things” One concern for this Dl characteristic was that 
novice users might still be confused and that there needed to be more information to guide these 
users: “In my experience, a lot of faculty members really don’t even understand what assessment 
is. i mean they have so little knowledge where their glossary isn’t going to be sufficient for them, at 
least that’s been my experience. Now, maybe people going into this website already know something 
about assessments, but i wouldn’t count on it.” 

Complexity - The main area of concern was that the information available on ASSESS was too 
confusing. Symbols and terminology used throughout the website were misunderstood and ques¬ 
tioned by participants: “Utility standards of program evaluation; I don’t even know what that means.” 
Another issue was that the amount of information provided was overwhelming for some participants, 
“it might just be ' We’re putting all this information out there for you to use,’ but I’m just wondering 
if it could be filtered down first so i can dick and go into more details, try to find in the areas that 
I’m more interested in, whatever it may be in those particular areas.” 

In order to minimize confusion, the development team reviewed the terminology used on 
the website. Some symbols and terms were altered and a glossary was added to assist with the 
remaining terminology. Additionally, a “learn” tab was added to the website. This tab includes 
information about the website, about assessment instruments, methods, practices and results, 
and the glossary of terms. While some participants had mentioned that the amount of informa¬ 
tion was troubling, the additional information will be sorted into smaller chunks. Therefore, the 
information would presumably not be overwhelming. Another feature that has been added to 
minimize confusion is a question mark icon available above potentially confusing sections of 
text. If this icon is hovered over using a mouse, a pop-up window opens with additional explana¬ 
tory information. These changes should make the information on the ASSESS website more 
applicable for all users. A video detailing information available on ASSESS can be accessed by 
clicking Figure 2. The video highlights the areas that participants noted as positive as well as 
the areas of improvement. 
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■ASSESS 

Engineering Education 

INSTRUMENT DETAILS 

Engineering Design Self-Efficacy Instrument 

Viewed 470 Times 

Back to Search Page 

Summary Information 

The Engineering Design Self-Efficacy Instrument, developed by Carberry and the Tufts University Center for Engineering Educational Outreach (2009). was designed to measure individuals' self- 
concepts toward engineering design tasks. The instrument includes 36 items that ask about students perceived levels of confidence, motivation, success in performance, and apprehension in 
performing engineering design process tasks. This instalment was designed to be in accord with the eight-step engineering design process established by the Massachusetts Department of Education 
(2001/2006). The first item in each section asks about the respondent's self-conception toward conducting engineering design, while the last eight items report on each step of the engineering design 
process, including: (1) identifying a design need, (2) researching a design need, (3) developing design solutions, (4) selecting the best possible design. (5) constructing a prototype. (6) testing and 
evaluating a design, (7) communicating a design, (8) and redesigning (Carberry Lee. & Ohland, 2010). The instrument possesses desirable properties that are appealing for use with college students, 
such as being relatively short, simple to administer, and requiring less than 10 minutes to complete. 

• Each instrument includes tabs of information. The summary tab gives the user an 
overview of the instrument including recommendations for use and how it has been 
used in engineering education. 

• General information about assessment instruments and terminology is also available. 

Figure 2. Available Information in ASSESS. Link: https://www.youtube.com/watch?v=JOupB 
71Avh4&index=7&list=PL84VcUp5xaBsRutUS-8KR2lwFxh8Kfl9G 

Community Features 

This category is focused on the interactive aspects of the website, including user ratings of assessment 
instruments and users’ ability to submit information about assessment instruments. Relative Advantage has a 
medium positive rating while Compatibility anti Observability are low positive ratings, as shown in Table 3. 

Relative Advantage - Including interactive possibilities on ASSESS increases the Relative Ad¬ 
vantage over alternative ways of researching assessment instruments. Participants felt that these 
features would be beneficial: “it becomes a very useful way for me to follow what others are doing” 
and “ Once it gets more reviews and commentary from the community i think you’re gaining collec¬ 
tive knowledge, gaining access to collective knowledge” 

Compatibility - Participants provided positive comments about the Community Features, because 
it gave them some control over the database. It was evident that participants wanted to have the 
ability to give input on the database: “The ability to have input on things by either offering reviews 
or submitting instruments yourself is good, that way it can grow, but grow in a way where there’s 
some control with reviews, maybe some control in quality.” 

Observability - The Community Features also provide participants with greater confidence in using 
ASSESS. As one user stated, “if I see a bunch of user comments then that tells me right away people 
are using this website” Participants expressed greater likelihood to use ASSESS if they knew other 
people were successfully using this resource. The Community Features demonstrate the number of 
users on ASSESS to potential users. 


Summary 

Description 

Availability 

Technical 

Specifications 

Reviews 

References 
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Originally the development team was uncertain as to whether to include user or expert reviews of 
assessment instruments on ASSESS. The feedback in the Community Features category convinced 
the developers that user reviews are a valuable feature to include in ASSESS. At this point, the ASSESS 
database includes only the functionality for user reviews, but expert reviews functionality may be added 
as well. Figure 3 provides access to a video about the Community Features in ASSESS. The video shows 
how users can rate instruments, the website itself, as well as proposing instruments for inclusion. 


Propose an Addition to Our Website Database 

Propose instrument candidates for the ASSESS database 

Instrument Information 
Instrument Name: 

Outcome or Construct Evaluated: 

Other information: 

I_J 

Source: 

Contact Information (of instrument owner) 

Name: 

Phone Number: 

Email: 

More Details 

□ Click here if you would like to add additional details about this instrument 

Submit 

49999 ^ 

Type the text _ I Privacy & Terms 


Submit Proposal 


• ASSESS is a user-driven database and there are opportunities for users to provide 
input on ASSESS. 

• Users are able to review instruments, propose instruments for inclusion, and propose 
instruments for development. 




Figure 3. Community Features in ASSESS. Link: https://www.voutube.com/watch?v=vVpmv 
Vmibz8&index=8&list=PL84VcUp5xaBsRutUS-8KR2lwFxh8Kfl9G 
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Use as an Alternative 

This area was of particular interest, as the developers’ priority was to ensure that the database 
would be a superior alternative to other methods for locating information about engineering edu¬ 
cation assessment instruments. 

Because of its significance, many interview questions were targeted to specifically address this area. 
Questions such as “What current methods do you use to access assessment instruments?” and “What 
advantages or disadvantages does this database have over your existing methods for finding assessment 
tools?” were used to develop an understanding of participants’ perspectives. Responses related to this 
category were found related to all Dl characteristics except for Complexity, as shown in Table 3. 

Relative Advantage - The most significant advantage over other methods was that ASSESS is a 
centralized location for information about assessment instruments. Many participants were frustrated 
with looking for information in a multitude of places and were looking for one inclusive resource: “Its 
relative advantage if it is completed as i think it's envisioned is it would be a single spot to collect 
data from a lot of different sources [in this case data about other assessment instruments! and it 
would reside in one spot.” 

Compatibility - Participants felt that ASSESS would meet their expectations for finding informa¬ 
tion about assessment instruments. It was also evident that participants felt that ASSESS would 
be the first resource of its type: “I’m intrigued with this concept in general because I think that this 
is something that we really need to have in the community.” Participants also agreed that ASSESS 
would assist them in identifying what assessment instrument they should use, “ifyou’re looking for 
tools you shouldn’t have to create them yourself like i do right now.” 

Trialability - Users reported that this Dl characteristic was high, because it is a feasible alterna¬ 
tive and requires no commitment to use. Participants were pleased that to use ASSESS there is no 
required login information and that it was free. The Trialability of ASSESS as compared to alterna¬ 
tives is excellent: “i mean as long as it’s free there’s no commitment. So, it makes it incredibly easy 
to commit to use it,” and “i certainly get the sense that with ASSESS i can find the things i need and 
look at it without having to commit to it.” 

Observability - Participants recognized how easy it is to refer others to ASSESS. Multiple par¬ 
ticipants mentioned that other methods of looking up information are not conducive for referring 
other people to, especially in the case of novices. However, in ASSESS, participants felt that the 
set up and collectiveness of the information made it easy to refer others to, including novices and 
students. For some participants, they were most excited about this potential advantage, “Andso, i 
definitely can see referring people to this website.” 

The overwhelmingly positive responses in the Use as an Alternative category were very promis¬ 
ing for the overall adoptability of ASSESS. If the ASSESS website is useful compared to existing 
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Instrument 0 
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Uses in 0 
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Reliability 0 

Validity 0 
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Biology Attitude Scale * 

0 
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Calibrated Peer Review 
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• 

0 
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California Critical Thinking Dispositions Inventory 

• 
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• 

Rate 

California Critical Thinking Skills Test 

• 

• 

• 

Rate 


• ASSESS is a single collection of information for engineering education assessment 
instruments. 

• All the information is in the same format and easily sorted and compared. 


Figure 4. Use of ASSESS as an Alternative. Link: https://www.youtube.com/watch?v=PlwlW 
hCacqO&index=9&list=PL84VcUp5xaBsRutUS-8KR2lwFxh8Kfl9G 


alternatives for potential users, adoption should increase. All four Dl characteristics that received 
responses for this category demonstrated that participants did find ASSESS useful as an alternative 
means of finding assessment instrument information. Because they were all positive, the develop¬ 
ment team made no changes based on this category of responses. A video highlighting the uses 
of ASSESS as an alternative is available by clicking on Figure 4. The main features identified by 
potential users as beneficial over alternatives is the focus of the video. 

Searchability 

Searchabi/ity is related to how easily and efficiently participants were able to use the search 
features in the database. These features include the key word search and the search categories 
available for narrowing a search. 

The ability to search the database is a necessary feature for potential users to effectively use the 
database. Participant responses on searchability are related to the Compatibility, Complexity, and 
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Trialability Dl characteristics. Compatibility received a medium positive rating, Complexity a low 
positive rating, and Trialability a low negative rating, as shown in Table 3. 

Compatibility - Participants found the database to be compatible for this category. They mentioned 
that most users should have experience using online search features and that ASSESS was similar to 
what they were used to. Being able to narrow their search using the side bar of options was also com¬ 
patible with their experiences and expectations, “I thought the search terms were fabulous.” The one 
concern for Compatibility in the Searchabi/ity category was with the engineering fields in the search 
limiting options. Some participants did not find their specific type of engineering in this field and thus 
were concerned that they would not be able to find information that would be helpful to them. 

Complexity - For the most part participants believed that the search features were not complex to 
use: “it seemed pretty straightforward as far as searching on the different fields.” A few participants had 
issues modifying the search fields after their initial modification. While a system was in use to modify 
the search, some participants did not understand how to use it. An additional source of confusion 
for some participants was the amount of narrowing terms available. Users can narrow their search by 
format, administration options, and technical aspects. Some participants mentioned that there were 
too many choices and that it was hard to know what they should use to narrow their search. 

Trialability - While this Dl characteristic did receive a negative rating, Trialability did not have 
many responses addressing Searchabi/ity. The only concern was over how the Searchabi/ity would 
change with the addition of instruments. Some participants felt that Searchabi/ity could change, 
specifically that it would decline, once the amount of instruments was larger. 

In order to improve ASSESS’ Searchabi/ity, the developers made a few adjustments to the search 
page and functions. To minimize confusion, the narrowing options are now accessed by clicking 
“Advanced Search.” Once this is clicked, three headings are shown that expand to show more 
options. The fields are also grouped by type, making it easier to find the correct field to narrow a 
search. An additional method of modifying search terms was added to address issues found by some 
participants: a “Show AN” button was added to the search page. This button allows users to browse 
the available information without performing a search, aiding those users who do not want to limit 
instruments they see. These improvements should assist in increasing Searchabi/ity for potential users. 
An explanatory video of the searchability features in ASSESS is available by clicking on Figure 5. 
The changes made based on participant feedback are presented in the video. 

Functionality of Design 

Functionality of Design includes perceptions of the functionality of the website. The focus is on 
how well ASSESS works as a database and as a website. Perceptions of the information and the 
usefulness of the database are not included. 
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Advanced Search 

instruments must have all of Assessment Search Engine 

me selected items to show up Search for instruments meeting your needs 

in results . J J 


Search 


Technical Aspects... 

Format Information... 

Administration 

Information... 


Search 


Search Show All 


Search in : m Item Names m Item Summaries m Item Tags 

Search Tips 


111 Results Found 


Instrument 0 


• ASSESS provides an efficient way to search for information. 

• Users can search utilizing a keyword search, a “show all” feature, and the advanced 
search fields. 


Figure 5. Searchability in ASSESS. Link: https://www.youtube.com/watch?v=-rBJiqv7i7s&in 
dex=10&list=PL84VcUp5xaBsRutUS-8KR2lwFxh8Kfl9G 


The Functionality of Design received varied responses from participants, but responses relating 
to this category were found in all five Dl characteristics. Table 3 shows that Relative Advantage and 
Complexity received medium positive ratings, Compatibility received a low negative rating, Trial- 
ability a low positive rating, and Observability a medium negative rating. 

Relative Advantage, Complexity, and Triaiabiiity - The responses for these Dl characteristics were all 
similar in nature. Participants felt that the website and database followed conventions with which they 
were familiar: “it is very simple to use and i think it would be easy to go back to and somebody wouldn’t 
walk away going, ‘Okay, I’m never doing that again’!’ For the Relative Advantage Dl characteristic partici¬ 
pants felt that the website was especially effective compared to alternative methods: “Then a definite 
upside is being able to find some of the best practices, the best tools with very little effort 

Observability - Participants were concerned that people might not be able to find the website: 
“i don’t know how e/se people would find it other than I got an email from you.” The main source of 
dissatisfaction for the Compatibility Dl characteristic was that it was unclear how to get to the as¬ 
sessment instruments. While ASSESS’s purpose is only to house information about the instruments 
and not the instruments themselves, some participants were expecting to see at least part of the 
instrument on the website. Because the instruments will not be available directly on the website, 
this means that users will have to use the available information to find the instrument. For some 
participants this was frustrating, while others had expected this. 
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Because some participants felt that the website was not going to be easy to find, the develop¬ 
ment team has worked on a publicity plan for ASSESS. In these efforts to spread the word about 
ASSESS, an emphasis on the purpose of the website will be made. Emphasizing that instruments 
will not be included in the website should ensure that potential users can develop correct expecta¬ 
tions of ASSESS. 

Diffusion of Innovations Characteristics 

Table 3 shows that the Relative Advantage and Complexity were the only Dl characteristics to 
receive high negative ratings in any category. Also noticeable is that Compatibility was the only Dl 
characteristic that had responses in all six categories. Therefore, these three Dl characteristics were 
considered the most important to focus on during development. Trialability and Observability overall 
did not have very many responses. While they each received one category with a high rating, this 
rating was achieved from an overall lower number of responses than the other three Dl character¬ 
istics received. Very few responses dealt with the Trialability and Observability Dl characteristics. 
Thus, Observability and Trialability were not deemed important. 


LESSONS LEARNED ON ADOPTION CHARACTERISTICS 

In a review of literature dealing with the adoption of technological innovations for use in education, 
predictions were that the Dl characteristics most important for ASSESS were Relative Advantage , 
Trialability, and Compatibility (Kebr\tdr\\ 2010, Martins et al. 2004, Samarwickrema and Stacey 2007). 
Most innovations, especially technical innovations, have Relative Advantage as a significant factor 
affecting their adoption. “ Diffusion scholars have found relative advantage to be one of the strongest 
predictors of an innovation’s rate of adoption” (Rogers 2003, pg. 233). Trialability was also predicted 
to be a significant Dl characteristic. Technological innovations often involve unfamiliar features or 
functions, and individuals tend to need a testing phase with the innovation before they are willing 
to fully adopt. Lastly, Compatibility was identified as likely to be an important Dl characteristic. The 
researchers believed that in order for an individual to adopt ASSESS, it would need to meet their 
expectations and work with their previous experiences. Also, if ASSESS did not fit the values or 
requirements of the individual’s institution, they would be less likely to adopt. 

As expected, Relative Advantage and Compatibility were both found to be important Dl charac¬ 
teristics for ASSESS. ASSESS has consistent information for assessment instruments, is searchable, 
and allows for user input. These features contribute to the Relative Advantage of ASSESS over 
alternatives. At the same time, these features are Compatible with users’ expectations, needs, and 
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experiences: “It’s very tailored to engineering education which is a fairly relatively speaking a new 
field, an emerging field. So that’s fantastic” 

Observability was not expected to be important for the adoption of ASSESS. This prediction was 
correct. This Dl characteristic did not receive very many responses. Many participants did not give 
it much importance: “ And then the Observability I’d say is still kind of unclear since it’s still in the 
development process,” and “ Just because it’s so new that it hasn’t had the opportunity to be visible. 
I’m not sure you’ll know the visibility also until it gets launched.” 

While Trialability was expected to be important, there were not very many responses that dealt with 
this Dl characteristic. ASSESS is currently free and does not require registration to search for instrument 
information. As such, Trialability is fairly high: “i mean that seems like it’s a high degree of trialability 
because it doesn’t seem like too much effort to go in there and find something that might be useful 
and learn a little bit about it and figure out whether it’s going to be what you need or not.” There may 
not have been very many responses about Trialability, because it is obviously high for ASSESS. 

Complexity was determined to be an important Dl characteristic, but had not been expected to 
be important. Many participants were concerned that novice users of assessment instruments would 
be confused by the information in ASSESS: “ I’m thinking about my friends who, they’re interested in 
incorporating engineering education kinds of things into their regular grants or what they’re trying 
to do in their classrooms or things like that but wouldn’t be able to distinguish between ... a more 
appropriate or a less appropriate instrument for what they want.” The ASSESS developers reviewed 
and adjusted terminology and formatting to decrease Complexity. 

The researchers believe that these results can be applied to other adoptability studies looking at 
technical innovations in engineering education. While ASSESS is specifically designed to house infor¬ 
mation for assessment instruments, the function of this innovation could be changed and still have 
the same general implications in regards to Dl characteristics and adoptability. Innovations meant to 
share curricula or exam questions between engineering educators would be expected to have similar 
results. Innovations that allow for user input are web-based, and serve as collections of information 
would be ideal applications for these results, regardless of the type of information involved. 


CONCLUSION 

The six categories identified during the analysis can be applied to the development of other online 
innovations as well. In looking at the responses in each category, Use as an Alternative and Func¬ 
tionality of Design seemed to have received the most attention from participants. In other research 
efforts, developers could consider focusing their efforts with potential adopters towards these two 
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categories. High ratings in Use as an Alternative are necessary for adoption, if the innovation cannot 
be satisfactorily used instead of other methods, it will most likely not be adopted. Similarly, high 
ratings in Functionality of Design are also important. If an innovation does not function acceptably 
for potential adopters, they are not likely to adopt. 

Relative Advantage, Complexity, and Compatibility were chosen as the most important Dl char¬ 
acteristics. If ASSESS offers no advantages, whether economic, social, efficiency, or other, then 
individuals have no reason to adopt it or other technological innovations. Compatibility and Com¬ 
plexity relate to Functionality of Design. The innovation cannot be difficult for adopters to use and 
it must be based on what is already familiar. Innovations are less likely to be adopted when they are 
too confusing or against the experiences of the user. 

The ASSESS development team is refining the website appearance and functionality to address 
issues of search effectiveness, information accuracy, and ease of use. The team continues to enter 
information for instruments already catalogued and to catalogue additional instruments as they are 
identified. ASSESS can now be found at: http://assess.tidee.orgl 
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